1. Field of the Invention
The present invention relates to a calibration technique of camera intrinsic parameters and a relative position and orientation between cameras.
2. Description of the Related Art
Conventionally, a virtual reality (VR) system is known. This VR system presents a three-dimensional computer graphics (CG) image created by a computer to the user to make him or her feel as if a virtual space were a real space. In recent years, a technique for presenting information which does not exist in real space to the user by synthesizing a three-dimensional CG to an image of real space has been developed. This technique is called a mixed reality (MR) system. This MR system aims at coexistence of the for a real space and a virtual space to coexist that can be experienced in only a condition isolated from the real space, and has received a lot of attention as a technique for enhancing virtual space.
As a typical apparatus that implements the aforementioned system (VR system/MR system), which allows the virtual space and real space to coexist, a head-mounted display (HMD) is available. The VR/MR system can present the virtual space and MR space to the user who wears the HMD by displaying the virtual space and real space on the HMD (see Japanese Patent Laid-Open No. 2002-159019). As an apparatus of this type, a stereo video see-through HMD (to be simply referred to as an HMD hereinafter) is used. The HMD has an outer appearance shape similar to spectacles or goggles, and right- and left-eye display devices including, for example LCDs are arranged inside the HMD. When a stereo video image is to be acquired by video cameras mounted on the HMD, right- and left-eye stereo video cameras are arranged at, for example, positions adjacent to the right and left viewpoint positions of the user (who wears the HMD).
In order to provide high-quality MR, accurate registration is required between the real space and virtual space. In the case of the aforementioned system using the HMD, the positions and orientations of both the right- and left-eye stereo video cameras on the real space are calculated, and three-dimensional CGs are rendered based on the respective positions and orientations and are synthesized to an image of real space. The positions and orientations of the cameras may be calculated from the values of sensors such as magnetic or optical sensors to the cameras. However, the relative position and orientation between the two cameras may often fail to be constant due to errors of the sensors, thus making the user feel unnatural. As a method of avoiding such unnaturalness, a method of using the relative position and orientation between the two cameras, which are calculated in advance, is effective. With this method, one of the right- and left-eye stereo video cameras is set as a main camera, and the other is set as a sub camera. Then, a sensor is attached to only the main camera to calculate the position and orientation, and the position and orientation of the sub camera are calculated using the position and orientation calculated using the sensor, and the relative position and orientation between the two cameras, which are calculated in advance.
Not only in the system using the HMD, camera intrinsic parameters have to be taken into consideration so as to attain accurate registration. The intrinsic parameters include, for example, a focal length, principal point position, and lens distortion. As for calculations of the intrinsic parameters, a method using a calibration pattern (see Japanese Patent Laid-Open No. 2005-017286), a method calculating the intrinsic parameters from a moving image (see Japanese Patent Laid-Open No. 2000-268179), and the like are known.
The aforementioned position and orientation between the two cameras and the camera intrinsic parameters are unique to each HMD. However, the camera intrinsic parameters and the relative position and orientation between the two cameras often change when the HMD is inadvertently dropped or due to arbitrary external factors. In such a case, they have to be calculated again. However, the user may continue to use the HMD depending on his or her knowledge or experience level with respect to the HMD or MR technology. When the user continues a job while wearing the HMD in such state, a normal stereoscopic viewing disturbance may adversely affect the body of the user.